File storage system

1 Introduction and Goals

This document provides an architectural overview of the File Storage Service (FSS), a solution designed to store and manage files, such as images, videos, and other media, for external applications.

1.1 Requirements Overview

Features:

  • Upload, download, and delete files via RESTful APIs.
  • Metadata management for stored files.
  • Access control mechanisms to manage file visibility and permissions.
  • Versioning for stored files.
  • Storage files on local file system or in AWS S3.

2 Architecture Constraints

Constraint Description
API Protocols The service must use REST APIs
File Storage Mechanism Files must be stored using an object storage solution on local files system or AWS S3

3 Context and Scope

Sunday, December 29, 2024 at 5:20 AM Central European Standard Time[System Context] File storage systemIdentity and AccessManagement (IAM)[Software System]-File storage system[Software System]-Person[Person]-External System[Person]-Actor[Person]-Take keysRemove link.Link options.ExtendsRemove link.Link options.ExtendsRemove link.Link options.AuthenticateRemove link.Link options.UsesRemove link.Link options.
PersonSoftware SystemRelationship

4 Solution Strategy

Goal Description
Security OAuth2 for secure user/externalSystem authentication

5 Building Block View

Diagram from 3.1. section displaying all systems:

Sunday, December 29, 2024 at 5:20 AM Central European Standard Time[System Context] File storage systemIdentity and AccessManagement (IAM)[Software System]-File storage system[Software System]-Person[Person]-External System[Person]-Actor[Person]-Take keysRemove link.Link options.ExtendsRemove link.Link options.ExtendsRemove link.Link options.AuthenticateRemove link.Link options.UsesRemove link.Link options.
PersonSoftware SystemRelationship

Block Description
File Storage System Main system responsible for storage files.
Identity and Access Management (IAM) system System responsible for authentication and user management.
Person Person can use REST Client to manually sent file.
External System Other system which store files.

5.1 Level 1 - File Storage System

The system architecture consists of two main components File Storage Service and supporting infrastructure.

Sunday, December 29, 2024 at 5:20 AM Central European Standard Time[Container] File storage systemIdentity and AccessManagement (IAM)[Software System]-File storage system[Software System]File storage service[Container: Spring]descriptionMongoDB[Container]-Kafka[Container]-Local File System[Container]-Actor[Person]-UsesRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.Take keysRemove link.Link options.AuthenticateRemove link.Link options.UsesRemove link.Link options.
ContainerPersonSoftware SystemRelationship

System or components Description
File Storage Service The service is implemented in Spring Framework
supporting infrastructure The system uses MongoDB as the primary database. Additionally Apache Kafka is utilized as a messaging system

To model the domain, the DDD (Domain-Driven Design) approach was applied, complemented by Event Storming. More information: Event Storming


5.1.1 Level 2 - File Storage Service

The service breaks down in 3 components:

  • container - component is responsible for containers management;
  • object - component is responsible for objects management;
  • storage - component is responsible for storing files;
    Sunday, December 29, 2024 at 5:20 AM Central European Standard Time[Component] File storage system - File storage serviceIdentity and AccessManagement (IAM)[Software System]-File storage service[Container]Container[Component]-Object[Component]-Storage[Component]-MongoDB[Container]-Kafka[Container]-Local File System[Container]-Actor[Person]-UsesRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.AuthenticateRemove link.Link options.UsesRemove link.Link options.UsesRemove link.Link options.
    ComponentContainerPersonSoftware SystemRelationship

5.1.2 File Storage Service - REST API documentation

Comprehensive documentation for the REST API can be accessed via the following link.


5.1.3 File Storage Service - Security

The service functions as a Resource Server. It utilizes scopes to determine whether the user has sufficient permissions to perform a specific action. The table below presents the basic roles required to obtain the corresponding scopes. For example, to obtain the scope fss.containers:read, the fss.containers-read role must be assigned.

Base role Scope Claims Access Granted
fss.containers-read fss.containers:read - Permission to read any container.
fss.containers-write fss.containers:write - Permission to create, change or delete container.
fss.objects-read fss.objects:read - Permission to read any object in container(PUBLIC or PRIVATE) and owns objects in PRIVATE containers.
fss.objects-write fss.objects:write - Permission to create, change or delete object in container(PUBLIC or PRIVATE) and owns objects in PRIVATE containers.
fss.objects-manage fss.objects:manage - Permission to read, create, change or delete any object in container.

Managing such basic roles individually would be cumbersome; therefore, two additional roles have been created to aggregate the basic roles. Role Mapping Matrix:

Role\Base role fss.containers-read fss.containers-write fss.objects-read fss.objects-write fss.objects-manage
fss.admin + + + + +
fss.user - - + + -

Example decoded token for user with fss.admin role:

{
  "exp": 1735338196,
  "iat": 1735337896,
  "auth_time": 1735336999,
  "jti": "4509bfe4-9436-4874-8a74-de511e9c0232",
  "iss": "http://localhost:8080/realms/f18",
  "aud": "account",
  "sub": "9bb82d81-f938-43ef-8aa8-c84706deb6db",
  "typ": "Bearer",
  "azp": "fss",
//....
  "scope": "openid fss.containers:write profile fss.objects:read fss.objects:write email fss.objects:manage fss.containers:read",
  "email_verified": true,
  "name": "Ala Kot",
  "preferred_username": "Ala Kot",
  "given_name": "Ala",
  "family_name": "Kot",
  "email": "ala@ma.kota"
}

The table below describe the permissions required to perform actions on objects in container.

method \ container type PUBLIC PROTECTED PRIVATE
fetch one object fss.objects-read owner or fss.objects-manage
fetch all objects fss.objects-read fss.objects-read fss.objects-manage
edit object fss.objects-write fss.objects-write owner or fss.objects-manage

5.1.4 File Storage Service - Container Components

Implemented in Layered architectural patter as Active Record.

The component enables:

  • creation,
  • modification, and
  • deletion of containers.

Each container is:

  • identified by a unique name
  • access types - define access of objects in container:
    • Public: All objects within the container are accessible to anyone without requiring authentication.
    • Protected: Objects are accessible only to authenticated actors.
    • Private: Objects are restricted to the object owner or actor with specific authority.
  • revision Policy - define revision policy for objects in container:
    • REVISION_ON - revision enable
    • REVISION_OFF - revision disable

Example container:

{
"containerId": "676f243e808bd90f188658a8",
"name": "custom_container_name",
"type": "PUBLIC",
"revisionPolicyType": "REVISION_ON"
}

5.1.5 File Storage Service - Object Components

Implemented in Port & Adapters architectural patter as Domain Model.

The component enables:

  • creation of objects,
  • modification of objects,
  • modification of objects by restoring previous object's revision if revision enable,
  • deletion of objects,
  • deletion of object's revisions if revision enable.

Example object when revision Policy is enabled(revision 3 was deleted):

{
  "objectId": "676f2666808bd90f188658a9",
  "key": "god.jpg",
  "containerId": "676f243e808bd90f188658a8",
  "revisions": [
    {
      "revisionId": 1,
      "revisionType": "CREATED",
      "size": 621697,
      "storageId": "676f2666808bd90f188658a9_1.jpg"
    },
    {
      "revisionId": 2,
      "revisionType": "UPDATED",
      "size": 621697,
      "storageId": "676f2666808bd90f188658a9_2.jpg"
    },
    {
      "revisionId": 4,
      "revisionType": "DELETED",
      "size": 0,
      "storageId": null
    },
    {
      "revisionId": 5,
      "revisionType": "RESTORED",
      "size": 621697,
      "storageId": "676f2666808bd90f188658a9_5.jpg"
    }
  ],
  "metadata": {
    "extension": "jpg",
    "ownerId": "9bb82d81-f938-43ef-8aa8-c84706deb6db"
  }
}

5.1.6 File Storage Service - Storage Components

Implemented in Layered architectural patter as Transaction Script.

The component is responsible for file storage in Local File System.


5.2 Level 1 - Identity and Access Management (IAM) system

The system integrates Keycloak as the Identity and Access Management (IAM) solution, eliminating the need to develop custom authentication and authorization mechanisms. This choice reduces development effort, enhances security, and ensures compliance with modern authentication standards like OAuth2 and OpenID Connect.

Sunday, December 29, 2024 at 5:20 AM Central European Standard Time[Container] Identity and Access Management (IAM)Identity and Access Management (IAM)[Software System]Keycloak[Container]-Keycloak Database[Container]-Actor[Person]-AuthenticateRemove link.Link options.UsesRemove link.Link options.
ContainerPersonRelationship

6 Runtime View

// TODO

7 Deployment View

7.1 DEV

Deployment diagram - dev:

Developer Laptop[Deployment Node]-File storage servicefrom File storage system[Container: Spring]descriptionDocker[Deployment Node]-MongoDBfrom File storage system[Container]-Kafkafrom File storage system[Container]-Sunday, December 29, 2024 at 5:20 AM Central European Standard Time[Deployment] DevelopmentUsesRemove link.Link options.UsesRemove link.Link options.
ContainerElementRelationship

7.1.1 IAM

  1. Run keycloak and postgresDB:

     docker-compose -p auth -f .\.docker\docker-compose-auth.yaml up -d
    
  2. Config realm

    1. Create realm using file realm-export.json
    2. Create user
    3. Regenerate Client Secret in fss Client

7.1.2 File Service

Run
  1. Setup dev env:

     docker-compose -p f18 -f .\.docker\docker-compose.yaml up -d
    
  2. Build everything:

    .\gradlew service:file-storage:build
    
  3. Start application:

     .\gradlew service:file-storage:bootRun
    
Test
  1. Execute tests:

     .\gradlew service:file-storage:test
    
Docker
Publish image
  1. Build image:

     docker build . -f .\.docker\file_storage_service_image\Dockerfile -t ghcr.io/grubeb/file-storage-service:latest --no-cache 
    
  2. Publish image:

      docker push ghcr.io/grubeb/file-storage-service:latest
    

8 Cross-cutting Concepts

8.1 Clean Architecture

Architecture is described in book: Get Your Hands Dirty on Clean Architecture, by Tom Hombergs.

9 Architecture Decisions

Link

10 Quality Requirements

11 Risks and Technical Debts

12 Glossary

Term Definition

13 attachments

13.1 System modeling

13.1.1 introduction

To model the domain, the DDD (Domain-Driven Design) approach was applied, complemented by Event Storming. Event Storming is an excellent tool for supporting DDD processes, particularly during the exploratory and design phases. It facilitates the identification of domain events, the modeling of business processes, and the development of a shared language among all stakeholders.

Event Storming is created using draw.io, and the corresponding file is located in the designated folder(./f18.drawio). Below, an exported image of the documentation is provided for reference:

13.1.2 Domain Description

The File Storage System is designed to manage and store files along with their associated metadata. It consists of two primary entities: Containers and the Objects they contain.

A Container is a logical grouping of Objects and serves as the primary organizational unit in the system. Each container is:

  • identified by a unique name
  • is assigned one of the following access types:
    • Public: All files within the container are accessible to anyone without requiring authentication.
    • Protected: Files are accessible only to authenticated actors.
    • Private: Files are restricted to the container owner or users with specific authority.
  • revision Policy:
    • REVISION_ON - revision enable
    • REVISION_OFF - revision disable

An Object represents a file stored in the system. Object consists of the following attributes:

  • Key: Derived from the file name,
  • Metadata: Includes additional information about the file, such as size, creation date, and custom attributes.

13.1.3 Strategic Design (ES: Pig Picture & Process Modeling)

During modeling, the following definitions were established as universal concepts, applicable across all bounded contexts within the domain:

  • container - logical container for objects;
  • object - represents a file stored in the system;

13.1.4 Legend:

img.png
img.png

13.1.5 ES. Pig Picture:

img.png
img.png

13.1.6 ES. Process Modeling:

  • container related processes

    img.png
    img.png

  • object related processes:

    img_1.png
    img_1.png
    img_2.png
    img_2.png

  • storage related processes:

    img_3.png
    img_3.png

13.1.7 Tactical Design (ES: Design Modeling)

img_4.png
img_4.png

After documenting and describing the processes, the following bounded contexts were identified:

1. container context

Implemented in Layered architectural patter as Active Record.

2. object context

Implemented in Port & Adapters architectural patter as Domain Model.

3. storage context

Implemented in Layered architectural patter as Transaction Script.